
Office of Naval Research 
Contract N00014-S7-A-0298- 00D6 MR- 372-012 

NATIONAL AERONAUTICS AND SPACE ADMINISTRATION 
Grant NCR 22-007-008 f 


SUFFICIENT CONDITIONS FOR NON-NEGATIVITY OF THE 
SECOND VARIATION IN SINGULAR AND NONSlNGUlAR CONTROL 
, - , . ; rm PROBLEMS * >V;, 


D.H.. Jacobson 


August 1909 


Technical Report No. 596 


[This ddcujn.e.nt has been apprbved" for public >>■ release 
und sale j _ its dist-rifcptibn is unlimited- , Reproduction in 
whole or in part i's pe rmitted ^by v tbd U . S*. ^Gpve rnxnent . 


Division of Engineering and Applied Physics 
harvard University * Cambridge, Massachusetts 



Office of Naval Research 
Contract N00014-67-A-0298-0006 
NR-372-012 


National Aeronautics and Space Administration 
Grant NGR 22-007-068 


SUFFICIENT CONDITIONS FOR NON -NEGATIVITY OF THE 
SECOND VARIATION IN SINGULAR AND NONSINGULAR 
CONTROL PROBLEMS 

By 

D. H. Jacobson 


Technical Report No. 596 


This document has been approved for public release 
and sale; its distribution is unlimited. Reproduction in 
whole or in part is permitted by the U. S. Government. 


August 1969 


The research reported in this document was made possible through 
support extended the Division of Engineering and Applied Physics, 
Harvard University by the U. S. Army Research Office, the U. S. 
Air Force Office of Scientific Research and the U. S. Office of 
Naval Research under the Joint Services Electronics Program by 
Contracts N00014-67-A-0298-0006, 0005, and 0008 and by the National 
Aeronautics and Space Administration under Grant NGR 22-007-068. 


Division of Engineering and Applied Physics 
Harvard University • Cambridge, Massachusetts 




SUFFICIENT CONDITIONS FOR NON-NEGATIVITY OF THE 


SECOND VARIATION IN SINGULAR AND NONSINGULAR 

CONTROL PROBLEMS 

By 

D. H. Jacobson 

Division of Engineering and Applied Physics 
Harvard University • Cambridge, Massachusetts 

ABSTRACT 

Sufficient conditions for non-negativity of the second variation in 
singular and nonsingular control problems are presented; these condi- 
tions are in the form of equalities and differential inequalities. Control 
problem examples illustrate the use of the new conditions. The relation- 
ships of the new conditions to existing necessary conditions of optimality 
for singular and nonsingular problems are discussed. When applied to 
nonsingular control problems, it is shown that the conditions are suffi- 
cient to ensure the boundedness of the solution of the well-known matrix 
Riccati differential equation; these conditions are less stringent than 
those known heretofore. 



I. Preliminaries 


1 . Introduction 

Singular control problems occur often in engineering; for example, 
in the aerospece industry a number of important problems are singular 
[ 1 ] , [2]. Mathematical economics is another field in which singular 
optimal control problems are common [3]. These and other examples 
have prompted researchers to inquire into the mathematical properties 
of singular arcs [4]-[20]. Circa 1964, Kelley [4] discovered a new 
necessary condition of optimality for singular arcs. This condition was 
generalized subsequently by Robbins [5], Tait [6], Kelley et al [7] and 
Goh [8], and is now commonly known as the generalized Legendre -Clebsch 
condition (or Kelley's condition). In [9] an additional necessary condition 
of optimality for singular arcs was derived and was shown to be non- 
equivalent to the generalized Legendre -Clebsch condition. For want of 
an alternative, we shall refer to this condition as Jacobson’s condition. 

In this paper we present sufficient conditions for non-negativity of 
the second variation in singular control problems; in strengthened form 
these conditions (equalities and inequalities) are sufficient for a weak 
relative minimum. Both Kelley's and Jacobson's necessary conditions 
of optimality are derived easily from the new conditions. We show that 
the conditions are applicable to totally singular, partially singular"^" and 
nonsingular control functions. Moreover, when applied to nonsingular 
problems, sufficient conditions for the boundedness of the solution of 
the well-known matrix Riccati differential equation are obtained; these 
are less stringent than those known heretofore [21], [26], 


-f Defined in Section I. 3. 
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Control problems without terminal constraints are considered 
first; the results are then generalized to the case where constraints on 
the terminal states are present. It turns out that the presence of ter- 
minal constraints does not complicate unduly the derivation. 

2. Problem Formulation 

We shall consider the class of control problems where the 
dynamical system is described by the ordinary differential equations: 

x = f(x,u, t) ; x(t Q ) = x q (1) 

where, (except for Section V): 

f(x,u, t) = f x (x, t) + f u (x, t)u (2) 

The performance of the system is measured by the cost functional: 

t f 

V{x o , t 0 ) = j L(x,t)dt + F(x(t f ),t f ) (3) 

^o 

and the terminal states must satisfy 

^ (x(t f ), t f ) = 0 . (4) 

The control function u(- ) is required to satisfy the following constraint: 
u(.)€U (5) 

where the set U is defined by: 

U s {u(- ) : |u.(t)| < l,t e [t 0 ,t f 3; i = 1, . . . ,m} . (6) 

Here, x is an n-dimensional state vector and u is an m-dimensional 

control vector, f^ is an n-dimensional vector function of x at time t and 

f is an n x m matrix function of x at time t; the functions L and F are 
u 

scalar. The terminal constraint function ^ is an s -dimensional column 
vector function of x(t^) at t^. The functions f, L, F and ^ are assumed 
to be three times continuously differentiable in each argument. The final 
time t£ is as sumed to be given explicitly. 
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The control problem is: determine the control function u(- ) which 
satisfies (4) and (5) and minimizes V(x o ,t Q ). 

3. Totally and Partially Singular Problems 

It can be shown that, along an optimal trajectory, the following 
necessary conditions (Pontryagin's Principle) hold: 

-*• = H x ( x ? u > x > t) ; Mt f ) = F x ^ x ^f)» V + ^ x v 

where 

u = arg min H{x, u, X, t) (8) 

ueU 


and 

H(x, u, X, t) = L(x, t) + X^f(x, u, t) (9) 

Here, x(- ), u(- ) denote the candidate state and control functions and X(- ) 

denotes an n-vector of Lagrange multiplier functions of time. 

In general the optimal control function (for the class of problems 

formulated in Section I. 2) consist of bang-bang sub-arcs and singular 

sub-arcs!^ A bang-bang arc is one along which the controls lie on the 

boundary of U and H , (x,X,t) / 0 j i = 1, . . . , m (except at a finite number 

i _ 

of switch times where the components of u change sign). 

A singular arc [17] is one along which 

H u (x, X, t) = 0 ; i=l,...,m (10) 

i 

for a finite time interval. 1 Note that this implies that, on a singular arc, 
H is independent of the control u. 


■j- ’Arc’ and 'sub -arc' are used synonymously. 

'f 1 For simplicity, we shall consider all the controls to be singular 

simultaneously. If this is not the case, no conceptual difficulties arise. 
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In the sequal we shall make use of the following definitions: 

Defn. 1 : A totally singular control function is one along which (10) 

holds for all t e [t , t,]. 

L o’ f J 

Defn. 2 : A partially singular control function is one along which (10) 

holds for k sub-intervals of length T\; i = 1, . . . ,k and where 
k 

I T i < <‘f - ‘o'- 
i=l 

II. Totally Singular Control Functions. Unconstrained Terminal State 
1 . Existing Necessary Conditions of Optimality 

In [7] Kelley et al show that the following (generalized Legendre - 
Clebsch) condition is necessary for the optimality of a singular arc: 

s 0 ( 11 ) 


where the q-th time derivative of is the first to contain explicitly the 
control u. Kelley et al used special control variations in order to derive 
this result; see [7], Recently an additional (Jacobson's) necessary con- 
dition was discovered [9]. In order for a singular arc to be optimal it 
is necessary that 


if T H 
2 u xu 


4h f 

2 ux u 


T 

+ f A Qf 
u u 




0 


( 12 ) 


where 

= H xx + f x Q + Q£ x 1 Q(t f> = F xx®‘ f >’‘ f ) < 13 > 

The partial derivatives f , H , H and f are all evaluated along the 
singular arc x(- ), u(* ). In [9] the above condition is derived for a scalar 
control using the technique of Differential. Dynamic Programming [22]; 
in that paper, Q(t) is shown to be the second partial derivative of V(x, t) 
with respect to x obtained whilst keeping u(- ) = u(* ). An alternative 
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derivation, using the Lagrange multiplier rule, is given in the Appendix 
of this paper. 

Of course, in addition to conditions (11) and (12), Pontryagin's 

Principle must be satisfied. 

2 

2. Second Variation (6 V) 

An expression for the second variation is [23]: 


a 2 v 


I 


{ j 6 x t h 

C. XX 


6x -f 6 u T H 


ux 


6x}dt + 


i6 x T F 


XX 


>x 


(14) 


o 


subject to the differential equation: 

6x = f 6x + f Su ; 6x(t ) = 0 . (15) 

In order for the singular (stationary) solution to be minimizing it is 
necessary that 

6 2 V 2* 0 (16) 

for all 8u(' ) sufficiently small to justify the second order expansion of 
V, and such that 

u(- ) + 6u(- ) e U . (17) 

Both Kelley's and Jacobson's conditions are necessary for (16) to hold; 
see [7] and the Appendix of this paper. In Section II. 4 we present suffi- 
cient conditions for (16) to hold. Note that the auxiliary minimization 

problem (14), (15) cannot be solved routinely because it is singular. 

*2 

3. Adjoining Linearized System to 0 Y 

We now adjoin (15) to (14) using a vector Lagrange multiplier 
function of time 6\(t): 


6 2 v = f {4 &x T H 6 x + 6u T H Sx + 5\ T [f 6xtf 6 U - 6k]} dt 
J 2 xx ux L x u 


+ i 6 x t F 6 x 

O XX 


(18) 
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T 

Integrating 6\ 6k by parts, we obtain: 

t. 


6 2 V = f { ^ 6 x T H 6 x + 6u T H 6 x + 6\ T [f 6 X + f 6 U ] 
2 xx ux L x u J 


+ 6\ i 6 x }dt + [^6 x T F 6 x - 6\’ A 6 X ] 

/. VV j 


XX 


( 19 ) 


Let us now choose 


6\(t) = 4-P(t)6x 


( 20 ) 


4 

where P(t) is an n x n symmetric', time varying, matrix. The second 


variation becomes: 


6 2 V 




6x T (P + H + f T P + Pf )6 X + 6 u T (H + f T P)6 x }dt 

XX X X ux u 


+ [^6x T F vv 6x - \ 6 x T P 6 x ] 


XX 


(21) 


subject to: 

6x = f 6x + f 6u ; 6x(t ) = 0 . (22) 

x u ’ o 

Note that 6 2 y = 6 2 Y, if (22) holds. 

2 

4. Sufficient Conditions for Non-negativity of 6 V 

As remarked in Section II. 2, the auxiliary problem (14), (15) or 
(21), (22) cannot be solved routinely owing to the fact that it is singular. 
Our new approach to the problem is to choose a bounded matrix function 
P(’) such that: 


-f There is no loss of generality in choosing P to be symmetric; this is 
so since if P were chosen to be unsymmetric, P + pT would appear in 
place of P in (21 ). 
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H + f T P = 0 Vt c [t ,tj (23) 

UX U 1 o’ f J 

Here, P(t) is an n x n symmetric matrix function of time and is 
m x n so that there are cases where (23) can be solved by choosing 
appropriate values for some of the elements of P. By choosing P ac- 
cording to (23) we annihilate the coefficients of the mixed 6u6x terms 
in (21). The remaining terms are quadratic forms in §x(t) and 6x(t^). 
Clearly, sufficient conditions for 5 V = 5 V ^ 0 are that (23) hold and 

P + H +f T P + Pf = M(t) 3* 0 (24) 

XX XX 7 

and 

-P(t f ) + F xx (x(t f ), t f ) = G(t f ) > 0 (25) 

Equality (23) together with inequalities (24) and (25) constitute sufficient 
2 

conditions for 6 V ^ 0, V 6x(‘ ). 

5. Sufficient Conditions for Optimality 

Sufficient conditions for a weak relative minimum are obtained by 
strengthening (24) and (25): 

P + H + f T P + Pf = M(t) > 0 Vte[t ,t f ] (26) 

XX X X O X 

-F(t f ) + F xx (x(tf),tf) = G(t f ) > 0 (27) 

To see this, note that if (23), (26) and (27) hold, then = 0 if 5x(’ ) = 0 
almost everywhere including t^. However, if 5x(- ) = 0 almost everywhere 
including t^, then by our assumptions on L and F -- see Section I. 2 -- we 
have that the total change in cost is: 


AY 



L(x + Sx, 



L(x, t)dt + F(x(t f ) + 6 x (t f ),t f ) 


- F(x(t f ), t f ) 


= 0 


( 28 ) 
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6 2 Y = 0 


AV = 0 


Thus we can always choose 6x(- ) / 0 sufficiently small so that 6 V is 
the dominant term in the expansion for AV; hence we have sufficiency. 
Example : H = 0, H > 0, F > 0. In this case, P(t) = P(t) = 0 
Vt £ and G(t^) = 0, satisfies (23), (26), (27). 

Note : If the dynamical equations (1) are linear and L and F are quadratic, 
then (23), (24) and (25) are sufficient conditions for optimality because 
all variations higher than the second vanish identically. 


Example : 


X 1 ~ x 2 


x 2 = u 


Xj (0) = 0 

x 2 (0) = 0 


V = j" 0 <I X 1 + 2 x 1 x 2 + 2 x 2> dt 


|uj < 1 . (3 

Here, u(- ) = 0 is a totally singular control which satisfies Pontryagin's 
Principle. We have that: 

H = 0 (3 

ux 

F = 0 (3 


Note that is not positive semi-definite. Equation (23) yields: 
P 1? (t) = P„(t) = 0 ; t e [0, 1] 


so that the left-hand side of (24) becomes: 
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“ • 



- 



- 


P 11 

0 

+ 

0 

P 11 

+ 

1 

2 

0 

0 _ 


- p ll 

0 _ 


2 

1 


and the left-hand side of (25) becomes: 
“-P n (t f ) 0" 

0 0 _ 


Inequalities (24) and (25) are satisfied if we choose 

* 11-0 ) 
p n = - 2 / 

and since the system dynamics are linear, and the cost quadratic, i 
is optimal. 

6. Relationship to Existing Necessary Conditions 

Both Kelley's [7] and Jacobson's [9] conditions can be derived 
from (23), (24), (25). 
a) Jacobson's Condition 
Let 

Q + P = P , 

where Q and P are both n x n, symmetric matrix functions of time, 
then, from (23), 

H + f T Q + f T P = 0 
ux u u 


so that 


if T (H + Qf + Pf ) + i(H + f T Q + f T P)f =0 
2 u xu u u 2 ux u u u 


From (24) and (40), 


-P - Q = H xx + f x (Q + P) + (Q + P)f x - M(t) 


(37) 

(38) 

(39) 
*) = 0 


(40) 

(41) 

(42) 


(43) 
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and from (25) and (40) 

-Q(t f ) - P(t f ) + F xx (x(t f ),t f ) - G(t f ) = 0 

Now set 

Q(t.f) - F xx ( x (t f ), t^) 


and 


so that 


-Q = H + f Q + Qf 

XX X X 


P(t f ) = -G(t f ) 


and 


-P = -M(t) + f T P + Pf 
x x 


Now, since 

M(t) >0 Yte [t Q ,t f ] and G(t f ) > 0 
we have that 

P(t) <0 vte [t Q , t f ] 

Using inequality (50) in (42), we obtain 

i f T H + jH f + f T Qf » 0 . 

2 u xu 2 ux u u u 


(44) 


(45) 


(46) 


(47) 


(48) 


(49) 


(50) 


(51) 


Inequality (51) together with (45) and (46) comprise Jacobson's necessary 
condition. 

b) Kelley's Condition (generalized Legendre-Clebsch) 

Differentiating (23) with respect to time yields: 

H + f T P + f T P = 0 = H + f T P - f T (H + f T P + Pf - M) = 0 

UX U U UX U U XX X x 


(52) 


Post multiplying (52) by f and adding its transpose, we obtain 

* • 1 i 1 HP 

H f +f H tf Pf if Pf - 2f x H f - 2f 1 f i Pf 

ux u u UX U U U U U XX U U X u 

- 2f T Pf f + 2f T Mf = 0 

U X u u u 


( 53 ) 
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Using 


in (53): 


T 

H = -f P 
ux u 


H f + f T H - f T H - H f - 2f T H f + 2f T f T H 

UX U U UX U XU UX U U XX U U X XU 


+ 2H f f + 2f Mf = 0 

UX XU u u 


(54) 


Rearranging (53), 


■2f T H f + 2H (f f - f ) + 2(f T f T - f T )H 

U XX U UX XU U U X U XU 


+ (H f ) + -J7 (f T H ) = -2f T Mf 
at ux u at u xu u u 


However, we have that 

M(t) ^ 0 Vt e [t , t f ] 


so that 


-f T H f + H (ff - f ) + (f T f T - f T )H 

U XX U UX XU U U X U XU 


+ (H f ) + i(f T H ) < 0 
2 dt ux u 2 u xu' 


Now, the left-hand side of (58) is just 


_ 9 _ 

9u 


dt 


2 u 


so that 


(- 1 ) 


9u 


dt 


? H 
2 u 


0 


(55) 


(56) 


(57) 


(58) 


(59) 


( 60 ) 


This is Kelley's first necessary condition. If this is met with equality, 
i. e. , 


f M(t)f = 0 
u u 


( 61 ) 



- 12 - 


then (56) is again differentiated with respect to time and (54) and (56) 
are substituted in. This yields Kelley's second condition, viz. , 


d_ 

8u 



> 0 


The generalized condition 

d^ 


( -i)q_9_ 
v ; au 


dt 


2q u| 


( 62 ) 


(63) 


is obtained by further differentiations. 

Note : In Section II. 5, we gave sufficient conditions for optimality; a 
requirement was that 


M(t) >0 Yt € [t Q , t f ] 


(64) 


However, this condition cannot hold unless q = 1 (see (56); if q > 1, then 
f^Mf^ = 0, contradicting (64) ). 


III. Totally Singular Control Functions. Constrained Terminal State 
1 . Second -Variation (S^V’fc) 

We shall allow the terminal constraint 

^(x(t f ),t f ) = 0 (65) 

where ^ is an s -dimensional vector function. As before, t^ is assumed 
to be given explicitly. 

If is adjoined to the cost functional by Lagrange multipliers 
v [23], the second variation is: 



o 


( 66 ) 
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subject to 
ancf^ 




5 x = f 5x + f 6u : 6x{t ) = 0 

x u ’ ' o 


6 X 

x 


= 0 


(67) 


( 68 ) 


2 

2. Adjoining Linearized System to 5 V* 

As in Section II, we adjoin (67) to (66) by a Lagrange multiplier 

T 

function 6\(- ). We integrate the term 5 X 6k by parts and set 


6\ = -|p(t)5 x 

We obtain finally: 


(69) 


t f 

5 2 V* = \ {y6 x T (P + H + f T P + Pf )6 X + 6 u T (H + f T P)6 x }dt 

2 ' XX X X ux u ' 


+ -|5x T (F xx + v T r// xx - P)6x| (70) 


subject to i^ x 5 x (t^) = 0. 

If xp has rank s, then s components of 5 x (t^) -- referred to as 
5 x S (t f ) -- can be solved for in terms of the remaining n - s components, 


5 x (t f ) ; for example: 


U 3 


where 


5 x s (t f ) •= -A 1 1 A 2 6 x n s (t f ) 


n 

s 1 [ S3 ] = ** 

s (n-s) 


(71) 


(72) 


-f More precisely, we have that (x + Sx')’ = f(x + 6 x ',u + 6u, t) and 
4* (x(t f ) + 5x , (t£),t^) = 0. However, expansions of these which are of 

higher-order than the first do not influence 5 2 V*. 

=f= If Aj is singular, then differently partitioned lp x and 5x(t^) must be used. 
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so that 




-A - ^ A 
1 2 

Sx n ' s (t f ) 


I 


6x(t f ) 


where Z is n x (n - s). 

We now eliminate the constraint 

6x| =0 

x 


6x n " s (t f ) = Z6x n ’ s (t f ) 


(73) 


(74) 


from (70) by using (73) in the boundary terms of 6 V*: 

tr 


5 2 Y = (* { i 6 x T (P + H + f T P + Pf )6x + 5 u T (H + f T P)6x}dt 

J, 2 ' xx x x ux u 


+ 4(Sx n-S ) T { Z T (F + v T ^ - P)Z}6x n " S 

2 ' XX XX 


(75) 


3. Sufficient Conditions for Non-negativity of 0 V 


Sufficient conditions for 6 V ^ 0 are (by analogy with Section II. 4): 


H + f x P = 0 
ux u 


P + H +f A P + Pf = M(t) > 0 

XX X X 


) Vt € [t o , t f ] 


and 


Z T (F + v T * - P)Z 

XX XX 


= G(t f ) > 0 


(76) 


(77) 


(78) 


Note that if s = 0 (no terminal constraints), 


n 


Z = nl I 
and (76)-(78) reduce to (23)-(25). 


(79) 
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4. Sufficient Conditions for Optimality 

By strengthening the inequalities in (77) and (78) we obtain 

S 2 V > 0 V6x(- ) t 0 (80) 

with 

6 V = 0 if 5x(- ) = 0 almost everywhere, including t^. (81) 

The argument of Section II. 5 can be used here to show that (80), (81) 
imply optimality (weak relative minimum). 

Note : As in the case of unconstrained terminal states these strengthened 
conditions can hold only if the singular arc is first-order (i. e. , the 
generalized Legendre -Clebsch condition holds with strict inequality for 

q= 1). 

5. Relationship to Existing Necessary Conditions 

As in Section II. 6 it is easy to show that satisfaction of (76)-(78) 
implies that Kelley's condition is satisfied. Jacobson's condition for 
problems with constrained terminal state is more complex than for the 
unconstrained case; see [9]. We shall not derive this condition here, 
from (76)-(78). 

6. Comment on Problems with Constrained Terminal State 

When deriving necessary conditions of optimality for problems with 
terminal constraints by constructing variations of the control function, 
one is faced with the task of showing that the chosen variation is indeed 
admissible [7], [9]. This is a formidable task even if the linearized 
dynamical system is assumed to be completely controllable and is 
assumed to have rank s."^ We remark that the approach taken in this 
paper does not require arguments of the type referred to above. We need 


-j- These are common assumptions [23]. 
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only assume that it is possible to satisfy ^ (x(t^), t^) = 0, and that ^ 
has rank s at x(t^), t^. We do not have to construct explicitly admissible 
control variations. 


IV. Partially Singular Control Functions ‘ 

1 . First and Second Variation (5V* + 5 2 y*) 

As defined in Section I. 3, a partially singular control function 
may have both singular and nonsingular portions (i. e. , sub-intervals 
of singular and bang -bang control). Along nonsingular arcs ^ 0, 
and the condition (Pontryagin's) 

min H(x, u, t) (82) 

ueU 

must hold (this is trivially satisfied along a singular arc). In this case 
the sum of the first and second variations is: 


Ov* + 5 2 v* = f {H T 6u + -y 6 x T H 6x + 6u T H 6 x }dt 

u 2 xx ux 


+ -y 6 x T (F + v T ty )5x 
2 xx xx 


(83) 


subject to 


6x = f 6x + f 6u : 6x(t ) = 0 

x u ’ o 


and 


i p Qx 

x 


= 0 


(84) 


(8 5) 


In order to enforce (85) (and ^ (x(t^), t^) = 0), we have 
u(’ ) + 5u(- ) e 


( 86 ) 


-j- In this section we treat the constrained terminal state problem; the 
unconstrained problem is a special case. 
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where 

U 2 =UOUj (87) 

and 

Uj s (u(-) :^(x(t f ),t f ) = 0, x = f(x, u, t); x(t Q ) = x q } (88) 

Note that by (82), 

H T 5u > 0 , u(-) + 6u(.) e U ? (89) 

with equality holding along singular arcs and at switch times of the 

bang-bang control arcs. If there are no singular arcs and no switchings 

of the control (i. e. , | | / 0, Vt e [t , t^] so that u(- ) = const. = + 1 or -1) 

then Pontryagin's Principle is sufficient for optimality because the 

second-order terms in (83) can be made insignificant (i. e. , dominated 

by 6u) for ||§u(* ) || sufficiently small. In the case where bang-bang 

arcs are present (i. e. where u(t) switches between its upper and lower 

bounds) one can, by placing a control variation in the immediate vacinity 

T 

of a switch point, cause 6u to contribute less to the change in cost 

6y s S + 6 V'* 5 than the second variation terms. 

2 

Clearly, sufficient conditions for 6 V* ^ 0 are (76)-(78) and suffi- 
cient conditions of optimality are these in strengthened form. Less 
restrictive sufficient conditions for purely bang-bang control functions 
have been given previously [24], [25]. However, in this section, we 
allow partially singular (i. e. , ’partially bang-bang’) control functions 
and thus embrace a wider class of problems than in [24] and [25]. 

V. Problems Nonlinear in Control 
1 . Introduction 

In the last section we indicated that our approach to sufficiency is 


independent of whether the control function is totally singular or partially 
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singular or, in the purely bang-bang case, nonsingular. In this section 
we study the more general nonsingular problem where the control u 
appears nonlinearly in f and L. We show that our approach is indeed 
applicable and give examples to illustrate our results. As a byproduct 
of the analysis, we obtain sufficient conditions for the boundedness of 
the solution of a certain matrix Riccati differential equation; these are 
less restrictive conditions than those obtained heretofore [21], [26]. 

We shall consider the following nonlinear optimal control problem: 
x = f(x, u, t) ; x(t Q ) = x q (90) 


V(x o’ t o ) = J L(x,u,t)dt + F(x(t f ),t f ) (91) 

^o 

Here it is assumed, for simplicity, that there are no constraints on 
the control u or on the terminal state x(t^), though this in no way limits 
the wider applicability of the analysis (see Section V. 5 for constrained 
terminal state). In this case the second variation is: 


6 2 V 


•r<* 


6x T H 


xx 


6x + 


6u T H 


ux 


6x + 


6u T H 


uu 


6u}dt 


+ 



F Sx 

XX 


t 


f 


(92) 


subject to 

6x = f 6 X + f 6u ; 6x(t ) = 0 (93) 

x u ’ o' 

Here, 

H (t) > 0 Vt e [t ,t ] (94) 

uu L o’ f 

is a well-known necessary condition (Legendre-Clebsch) of optimality. 
For the problem to be nonsingular, strict inequality must hold, i. e. , 
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H (t) > 0 Yt e [t , tJ 

nn *- o 7 ± J 


uu 


(95) 




A known necessary condition of optimality' [26] (which together with 
(95) and Pontryagin's Principle forms sufficient conditions of optimality) 
is that the solution to the following matrix Riccati differential equation 
be bounded for t e [^j^]: 

-S = H + f T S + Sf - (H +f T S) T H _1 (H + f T S) ; 

XX X X ux u uu ux u ’ 


S(t f ) = F 


XX 


(96) 


Sufficient conditions for the boundedness of S(* ) are known to be [21], [26], 


H 




XX XU UU UX 

F (*(t,),t f ) > 0 

xx f ’ i 

H _1 (t) > 0 
uu 




(97) 


vt e [t Q , t f ] 

2. Sufficient Conditions for Optimality 

Equation (93) can be adjoined to (92) using a vector Lagrange 
multiplier function of time §M' ). If, as before, we let 

S\ = ip(t)6 x 

then, the second variation becomes : 


(98) 


6 2 V = f { i 6 x T (P + H + f T P + Pf )6x + 5 u T (H + f T P)6x 
l 2 ' xx x x ux u 

n 

o 


+ 4 Q u T H 6 u }dt + ^5 x T (F - P)6x 
2 uu 2 xx 


( 99 ) 


-j- Classically known as the ’no -conjugate -point condition* [27]. 
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Clearly, 5 V > 0 if we can choose P(t) (which has bounded elements) 


so that 


H + f P =0 Yt e [t ,t.l 
ux u L o’ f J 


( 100 ) 


P + H + f P + Pf = M(t) >0 Yt e [t ,t r ] 

XX X V L n) fJ 


( 101 ) 


-P(tf) + F xx (x(t f ),t f ) = G(t f ) > 0 
Moreover, because of (95), we have that 
6 2 V > kN 2 [6u(- )] V6u(-) 


( 102 ) 


(103) 


where N is a suitable norm on 5u(* ) and k > 0. Equality (103) indicates 
that 0 Y is strongly positive and, by a theorem of Gelfand and Fomin 
[27, p. 100], this is sufficient for u(- ) to be a minimizing control func- 
tion (weak relative minimum). Thus conditions (100)-(102) are sufficient 
for optimality in this nonsingular problem. As an immediate consequence 
we have the following result: Conditions (100)-(102) imply that the matrix 
Riccati equation (96) has a bounded solution in the interval [t Q? t^] (because 
the boundedness of S(- ) is a necessary condition of optimality). These 
conditions are, in general, considerably weaker than (97), as the following 
example illustrates. 

Example : 


X 1 x 2 


x 2 = u 


(104) 


C , 1 2 ,12,1 2. 

j ^ 2 X 1 + 2 x 1 x 2 + 2 x 2 + 2 u ^ dt 


(105) 


Here, 


1 2 

H = 0 , H = 
ux ’ xx 2 1 


F = 0 


(106) 
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These values do not satisfy conditions (97). However, from Section II. 5, 
we have that: 

P 12 (t) = P 22 (t) = 0 = P ll (t) Vte[t o ,t f ] (107) 

and 

P u ^ "I (108) 

satisfy (100)-(102), so that the stationary solution to this problem, 
obtained from Pontryagin's Principle, is optimal. Note that in this 
particular case the checking of (100)-(102) is considerably easier than 
integrating the matrix Riccati differential equation to see whether or 
not its solution is bounded in the interval [0, 1]. 

3. Derivation of Riccati Equation 

The Riccati differential equation (96) can be derived directly from 
(100)-(102) as follows: From (100) and (101), 

-P = H + f T P + Pf - M(t) - (H + f T P) T H _1 (H + f T P) . 

XX X X ux u uu ux u 


(109) 


Let 


P = P + S 


(HO) 


then 


-P - S = H + f T (P + S) + (P + S)f - M(t) 

XX X X 


[H + f T (P + S)] T H _1 [H + f T (P + S)1 

L ux u J UU L ux u J 


(111) 


= H + f T (P + S) + (P + S)f - M(t) - (H + f T S) T H -1 (H +f T S) 

XX X X ux u uu ux u 

T T -1 T — — -1 T 

- (H + f S) H f P - Pf H (H + f S) 
ux u uu u u uu ux u 


— -1 T — 

- Pf H f*P 
u uu u 


( 112 ) 
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Using (100) and (110) in the last three terms of (112) we obtain: 

-P - S = H + f T (S + P) + (P + S)f - (H + f T S) T H _1 (H + f T S) 

XX X X ux u uu' ux u 

+ Pf H -1 f T P - M(t) (113) 

u uu u ' ' 

Now choose 

-P = -M(t) + f T P + Pf + Pf H _1 f T P . (114) 

X X u uu u v ' 

From (102) and (110), we have that 


-F(t f ) - S(t f ) + F = G(t f ) > 0 . (115) 

Choose, 

P(t f ) = -G(t f ) (116) 

Now we have that P(t) is bounded in the interval [t Q ,t^]. This follows 
from the fact that (-P) satisfies a Riccati equation for which conditions 
(97) hold, viz. , 

(117) 

G(t f ) » 0 


M(t) 


H _1 (t) > 0 
uu 


Yt e [t Q , t f ] 


Using these results in (113) and (115), we obtain finally: 


(118) 

which is the Riccati equation (96). Now since (100)-(102) are satisfied 
by a matrix function P(* ) which has bounded elements, and since, by (117), 
P(* ) is bounded, we have from (110) the result that S(* ) is bounded. 


-S = H + f T S + Sf - (H + f T S) T H~ 1 (H + f T S) 

XX X X ux u uu ux u 


s{t f ) ■ F xx (x(t f )>t f ) 
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4. Another Example 


X 1 " x 2 


x 2 = u 


; x 1 (o) = x 1Q 

; x 2 (0) = x 2Q 


V = ^ + 2x 2 + -^u^Jdt 


( 119 ) 


( 120 ) 


Here, 


H =0 
ux 


H 


XX 


n 

0 




F = 0 


( 121 ) 


1-1 

0 4J 

These values do not satisfy conditions (97). Conditions (100)-(102) 
become 

t P 12 P 22^ = 0 


P 11 0 


L o 


o 


LP 


li 


li 
o J 


-l o 

0 4 




(122) 


(123) 


and 


> 


0 


(124) 


-P u (i) o 

_o 0. 

Let us choose P^(lf) = 0} this satisfies (124). From (122), P^(t) = P 22 (t) = 0, 

Yt e [t , t,l. If we choose: 
o 7 ± 

P u = 2 and P u (0) = -2 , (125) 

then (123) becomes 

1 -2 + 2t~ 


-2 + 2t 


0 


(126) 
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Inequality (126) holds Vt £ [0, 1]. Thus the solution obtained from 
Pontryagin's Principle is optimal, and the Riccati equation associated 
with the above control problem has a bounded solution. 

5. Constrained Terminal State 

From Sections III. 3 and V. 2, sufficient conditions for optimality 


are: 


H + f x P = 0 
ux u 


Vte[t o ,tf] 


P + H + f P + Pf = M'(t) ^ 0 


Z i (F + v l Is - P)Z 

XX XX 


G'(t f ) > 0 


(127) 

(128) 
(129) 


and (Legendre-Clebsch) 


H (t) > 0 Vt e [t , t J 
uu L o’ f 


Example: 


Xi=x 2 Xl (0) = x 10 ; x x (l) = 0 


x z = u 


x 2 (0) = x 


20 


x 2 (l) = 0 


(130) 


(131) 


,, _ C , 1 2 1 2 1 2 

V_ J ^ 2 x 1 + 2 x 2 + 2 u 


)dt 


(132) 


Here, 


H = 0 
ux 


H 


XX 


-l 

0 


, F = 0, H =1 
’ ’ uu 


(133) 


(134) 


In this case, because n = s = 2, condition (129) disappears. As before, 
we have that 


H + f A P = [P 1? Poo] = 0 
ux u 12 22 J 


( 135 ) 
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Condition (128) becomes: 


" p ll 

o' 

+ 

0 

p ll 

+ 

-1 

o' 

_ 0 

0 _ 


- p ll 

0 _ 


_0 

1 _ 


Choosing P^(0) = -1 and Pjj = 2, the left-hand side of (136) 


( 136 ) 


becomes : 


1 -1 + 2t 

-1 + 2t 1 


(137) 


which is ^ 0 Vt e [0, 1], so that the stationary solution obtained from 
Pontryagin's Principle is optimal. Note that the above sufficiency condi- 
tions are, in this case, easier to check than the usual sufficiency condi- 
tions for nonsingular, constrained terminal state problems [23]. Moreover, 
the presence of the terminal constraints actually makes the choice of P 
and P(t^) easier (if in this example there were no terminal constraints, 
Inequality (129) would be violated by our above choice of P^(0) and P^). 


VI. Applicability of the New Conditions 
If the conditions 


H + f P = 0 
ux u 




Vt e [t o , t f ] 


P + H + f P + Pf = M'(t) » 0 

XX X X 


z t (f + v T ^ - P)Z 

XX XX 


- G'(t f ) > o 


(138) 

(139) 

( 120 ) 


cannot be satisfied, then no conclusion can be drawn regarding the nature 
(optimality or nonoptimality) of the stationary control function. This is 
because the above conditions are sufficient (but probably not necessary). 
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Example : 

x = u x(t ) = 0 

' o 

W « i 


V = 



1 2 A*. 

2 X dt 


(t f )x 2 (t f ) 


(141) 

(142) 

(143) 


Clearly, 

u(- ) = 0 (144) 

is a stationary solution for the above problem. Here, H = 0, H =1, 

UX XX 

F xx = -a(t^) and P is scalar so that (100) determines 

P(t) = 0 vtc [o, 1] . (145) 


Condition (101) becomes 

1 > 0 


(146) 


and condition (102) becomes 

-S > 0 . (147) 

Clearly, (101) is satisfied and (102) is satisfied if 

a(t f ) < 0 (143) 

but is violated if 

a(t f ) > 0 . (149) 

However, application of Jacobson's necessary condition [9] to this problem 
shows that if a(t^) > 0, the stationary solution (144) is not minimizing. 

The above example suggests the following sufficient condition for 
nonoptimality of a singular control function. 


VII. Sufficient Conditions for Nonoptimality of a Singular Control Function 
The second variation for the unconstrained terminal state problem 
is : 



6 2 V = \ {i6x T (P + H + f T P + Pf )Sx + 6 u T (H + f T P)6x}dt 
J, 2 XX X X ux u 


+ i6x T (F -P)5x| 
2 xx 


If it is possible to choose P(t)j t % [t , t ,.] such that: 


P + H + f P + Pf = M n (t) < 0 

XX X X 


-P(t f ) + F xx (x(t f ),t f ) = G"(t f ) < 0 


IT 1 T 

4f H +4H f + f Pf <0 ( 

2 u xu 2 ux u u u 

then the singular control is nonoptimal. 

The first two conditions cause the quadratic forms in Qx and 

6x(t^) in (150) to be nonpositive. If a rectangular pulse variation 6u(* ) 

of height rj and duration AT is introduced, then the dominant term (for 

r) and AT sufficiently small) of 


So that if 


I 

\ 6 u T (H + f T P)6 x dt 
J ux u 


ir) T [if T H +^H f + f T Pf ]rj(AT) 2 
2 ' L 2 u xu 2 ux u u u J ' 


IT 1 T 

if H + f + f Pf < 0 
2 u xu 2 ux u u u 


then 


6 2 V < 0 


and the singular control is not minimizing. 
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Example : 


x = u • x(0) = 0 

1 

V = J |x 2 dt - x 2 (t f ) 


(158) 

(159) 


In this case, conditions (151) and (152) become 

P + 1 < 0 => P < 1 (160) 

-P(t f ) -2^0 => P(t f ) > -2 (161) 

and 


4f T H + H f + f T Pf =P 
C U XU ux u u u 


( 162 ) 


Choose, 

P = -1 and P(t^) = -2 (163) 

then conditions (1 51) — (1 53) are satisfied and the singular arc is nonoptimal. 


VIII. Conclusion 

In this paper sufficient conditions are presented for the second 
variation to be non-negative in both singular and nonsingular control 
problems. It is demonstrated that known necessary conditions of opti- 
mality for singular problems and the no- conjugate -point condition for 
nonsingular problems are implied by the new conditions. Simple illustra- 
tive examples demonstrate the usefulness of the new conditions. A suffi- 
cient condition of optimality for singular problems is obtained by 
strengthening the inequality conditions; it is shown that these strengthened 
conditions can only be satisfied by first-order singular problems. 

When applied to the nonsingular control problem, the new conditions 
yield less restrictive sufficient conditions for the boundedness of the solution 
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of the matrix Riccati differential equation than were known heretofore; 
this result appears to be useful in its own right. 

The derivations presented are carried out for the case of u an 
n- vector, and s -vector constraints on the terminal state are permitted. 
Throughout, the final time t^ is assumed to be given explicitly; the 
generalization of the conditions to the case where t^ is given implicitly 
is straightforward but tedious. 

The appendix contains a Lagrange multiplier derivation of a neces 
sary condition of optimality for singular control problems which was 
derived previously using Differential Dynamic Programming [9]. 
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Appendix A: Lagrange Multiplier Derivation of Jacobson's Necessary 
Condition of Optimality for Singular Problems (No Terminal Constraints). 


The second variation is: 
t f 

S 2 V = f (4 6 x T H 6x + 6u T H §x}dt + 4 6x T F 6 x 
Jj. 2 xx ux 2 xx 


subject to 

6x = f §x + f 5u : 5x(t ) = 0 

X u ’ 'o' 

Adjoining (A. 2) to (A. 1) with Lagrange multiplier 
S\ = ^Q(t)6x 


(A. 1) 


(A. 2) 


(A. 3) 


(where Q is an n x n symmetric matrix function of time) and integrating 
by parts, we obtain 

t f 

S 2 V = f {4-6x T (Q+ H + f T Q + Qf )6 X + 6u T (H + f T Q)5 x }dt 
J 2 ' xx x x 7 ux u 


+ I^Xx - Q > S * 


(A. 4) 


Now, choose 


Q = H xx + £ x Q + Qf x ; Q <*f> = F xx<*<‘£>> *f) 


(A. 5) 


then, 


6 2 V = f 6 u T (H + f T Q)6xdt 
J ux u 


(A. 6) 


Introduce a variation 6u(- ) which is zero everywhere except, say, in the 
interval [t^,t^ + AT] where 

t^ and t^ + AT e [^tf] > 

and which has constant magnitude rj (note that u(- ) + 6u(- ) e U), 


(A. 7) 
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This inequality, together with (A. 5) comprise the necessary condition 
of optimality obtained (for the case of scalar control), using Differential 
Dynamic Programming, in [9]. 
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